Functional renormalization group for the anisotropic triangular antiferromagnet 
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We present a functional renormalization group scheme that allows us to calculate frustrated mag- 
netic systems of arbitrary lattice geometry beyond 0(200) sites from first principles. We study the 
magnetic susceptibility of the antiferromagnetic (AFM) spin- 1/2 Heisenberg model ground state on 
the spatially anisotropic triangular lattice, where J' denotes the coupling strength of the intrachain 
bonds along one lattice direction and J the coupling strength of the interchain bonds. We identify 
three distinct phases of the Heisenberg model. Increasing ^ = J'/J from the effective square lattice 
^ = 0, we find an AFM Neel order to spiral order transition at ^ci ~ 0.6 — 0.7, with indication to be 
of second order. In addition, above the isotropic point at ~ 1.1, we find a first order transition 
to a magnetically disordered phase with colhnear AFM stripe fiuctuations. 

PACS numbers: 75.10.Jm 



Frustration induced by lattiee geometry and interac- 
tion in two-dimensional quantum antiferromagnets con- 
stitutes a considerable challenge beyond mean field the- 
ory This is the main reason why the question of 
distinguishing systems with magnetic long-range order 
and spin liquid behavior has been one of the most diffi- 
cult and long-standing problems in the field of quantum 
magnetism. A plethora of different methods has been de- 
veloped to address this problem in the context of many 
different lattices and magnetic Hamiltonians, most of 
which, however, have severe drawbacks of very different 
kind. Quantum Monte Carlo (QMC) methods [3 may 
suffer from the sign problem often encountered for frus- 
trated systems, while exact diagonalization (ED) studies 
are constrained to small system sizes. Density matrix 
renormalization group (DMRG) methods [^Q] partly re- 
solve the latter problem, but are constrained to (effec- 
tively) one-dimensional systems. Coupled cluster meth- 
ods (CCM) and variational QMC methods [6[ are valu- 
able approaches to treat frustrated two-dimensional sys- 
tems. However, they are limited in the sense that, gener- 
ally, only trial state energies can be tested against each 
other, and no unbiased treatment from first principles is 
possible. 

Promising lines of improvement have been undertaken 
recently. In special cases where it is applicable, dimer 
projection schemes allow to treat larger systems at a 
similar level of precision and completeness as ED Q. 
From a perturbative expansion perspective, continuous 
unitary transformation (CUT) methods provide a help- 
ful tool to compute the energies of the ground state and 
excitation modes of small quasiparticle sectors d, [^. 
Furthermore, the generalized notion of matrix product 
states, in certain cases, leads to an efficient treatment 
of two-dimensional magnetic systems by projected en- 
tangled pair states (PEPS) In particular, the multi- 
scale entanglement renormalization Ansatz (MERA) pro- 



vides a new tool to compute energies and scaling behav- 
ior of certain magnetic systems, and has been most re- 
cently applied to frustrated systems Still, even in 
the optimal cases where the approximations made within 
these methods are controlled, it is mainly suited to deter- 
mine certain ground state properties only. For frustrated 
magnetic systems and actual comparison to experiment, 
however, it would be most desirable to compute comple- 
mentary thermodynamic quantities that allow to resolve 
the competition and classification of magnetic ordering 
and quantum fluctuations. The most suitable quantity 
in this respect is the magnetic susceptibility. Interpreted 
as the magnon spectral function, it provides detailed in- 
formation about the qualitative and quantitative type of 
magnetic fluctuations, and is the canonic quantity mea- 
sured in experiment. In this Letter, we employ the pseud- 
ofermion functional renormalization group (PFFRG) [isj 
as a new method to tackle systems of frustrated mag- 
netism. Applying the method to the anisotropic triangu- 
lar lattice, we demonstrate that the PFFRG is able to (i) 
treat large system sizes of 0(200) sites, (ii) is applicable 
to arbitrary frustrated lattice geometries and two-body 
bare interactions, (iii) naturally allows to compute the 
magnetic susceptibility as the canonical outcome of the 
RG, and (iv) hence provides an unbiased calculation from 
first principles that allows comparison to experiment. 

The Heisenberg model on the anisotropic triangu- 
lar lattice (ATLHM) attracted considerable attention 
in recent years as an experimentally accessible testing 
ground for quantum magnetism disorder phenomena. 
The Hamiltonian is given as 

H ATLHM ~ J ^ SiSj + J' ^ SiSj, (1) 

where the coupling J' applies to the bonds along (hor- 
izontal) one-dimensional chains and J is the coupling 
between them, forming a triangular lattice altogether 



FIG. 1. (color online) a) Triangular lattice structure. The 
horizontal bonds correspond to coupling J' the other to cou- 
pling J. b) Schematic plot of the hexagonal Brillouin zone. 
Different magnetic order resides at different points (shown: 
AFM Neel order, 120° Neel order, coUinear AFM order (c- 
AFM)). The open circles relate to the filled circles by recipro- 
cal lattice vectors, i.e., AFM order corresponds to two points 
in the BZ, coUinear AFM to four points, and 120° Neel order 
to six points. 



(Fig. [T^). We define ^ = J'/ J as a parameter to in- 
terpolate between the effective square lattice limit ^ = 
and the disordered isolated chain limit ^ — > oo. Experi- 
ments on CS2CUCI4 2.94) provide a good testing 
ground of discussing various features of spin liquid behav- 
ior. The formation of a magnetically ordered state for 
smaller ^ opposed to disorder tendencies for larger ^ can 
be nicely studied for the organic n - (BEDT - TTF)2X 
family. While X = Cu2[N(CN)2]Cl shows an AFM 
transition of Tn = 27K with estimated ^ = 0.55, the 
X = Cu2(CN)3 compound, estimated to be nearby the 
symmetric triangular regime ^ = 1.15, does not show 
magnetic order down to very low temperatures 15[ . De- 
tails of the phase diagram of ([1]) are still of current de- 
bate. It is an established fact that the system is AFM 
Neel-ordcred for small ^, changing to 120° Neel order 
as the isotropic triangular limit ^ = 1 is reached [l7- 23|. 
However, an intermediate disordered phase has been pro- 
posed |24{ . while other works assume a direct transition, 
but cannot classify the transition to be of second or first 
order [21I . For larger C > 1 : some works claim a dis- 
ordered phase extending to the ^ — )■ 00 limit [l^ l^sf . 
whereas others indicate coUinear antiferromagnetic (c- 
AFM) ordering [H,!!!!!^. 

We now address this problem with PFFRG, which is 
explained in more detail in 



131. Unlike conventional 



FRG studies, our starting point is not given by the 
bare excitations of the spin system. Instead, we use 
the pscudofermion representation of spin-1/2 operators 

- l/2Ea^/>a/j//3: -t,;, ^^ - x,y,z) with 

the fermionic operators /•]■ and /j, and the Pauli-matrices 
CT^ . This representation enables us to apply Wick's theo- 
rem leading to standard Feynman many-body techniques. 
The introduction of pseudofermions comes along with an 
artificial enlargement of the Hilbert space and therefore 
requires the fulfillment of an occupancy constraint (exclu- 
sion of empty and doubly occupied states) . Since an un- 
physical occupation acts as a vacancy in the spin lattice 



(c) 




(d) X = <3> 



FIG. 2. Graphic representation of the FRG differential equa- 
tions. Bare lines denote the (renormalized and scale depen- 
dent) Green's functions and slashed lines the single scale prop- 
agators. Eqs. (a) and (b) are the FRG equations for the 
self-energy and the two-particle vertex, respectively (without 
distinguishing between the different pairing channels on the 
right side of (b)). The Katanin scheme is given by the replace- 
ment (c). Fusing the external legs of the two-particle vertex, 
the magnetic susceptibility is obtained as shown in (d). 



associated with an excitation energy of order J, particle 
number fluctuations are suppressed at zero temperature 
and the constraint is naturally fulfilled. Quantum spin 
models are inherently strong coupling models, requiring 
infinite self-consistent resummations of perturbation the- 
ory. In this context FRG 27-33 provides a systematic 
summation in different interaction channels by generat- 
ing equations for the evolution of all one-particle irre- 
ducible m-particle vertex functions under the flow of an 
IR frequency cutoff A (see Fig. [5^ and 2b for the flow of 
the self-energy and the two-particle vertex). In order to 
reduce the infinite hierarchy of equations to a closed set, 
some truncation scheme has to be applied. As an impor- 
tant difference in the PFFRG as compared to conven- 
tional FRG schemes, we still include certain 2-loop con- 
tributions that are shown by Katanin (sij to be essential 
for a better fulfillment of Ward- identities (see Fig. [2};). 
In particular, this way the random phase approximation 
(RPA) is recovered as a diagram subset generated by the 
RG flow [i^l • It is important to emphasize that the RPA 
diagrams play a crucial role in our scheme as they are 
responsible for the collective ordering phenomena. In ad- 
dition, however, our method treats the leading diagram- 
matic contributions of a 1/5" expansion 35 . | 3q (direct 
particle- hole graphs) , 1/A^ expansion (33L 34l| (crossed 
particle- hole graphs), and particle-particle diagrams on 
the same footing. This enables us to adequately ac- 
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FIG. 3. (color online) Static magnetic susceptibility resolved for the whole Brillouin zone, varying ^ = J' j J from the effective 
square lattice ^ = 0.0 close to the triangular lattice ^ = 0.9. Bottom Left: 2-particle vertex flow for the AFM Neel channel 
(blue) versus the f20° Neel channel (red) at ^ = with respect to the IR cutoff flow parameter A. We observe that the AFM 
vertices start to diverge, signalling a magnetic instability. Susceptibilities are always given in units of f/J. The respective 
types of order for the distinct peaks positions are shown in Fig. [TJj. 



count for the competition of magnetic order and disor- 
der in an unbiased fashion. The magnetic susceptibility 
can be conveniently computed from the two-particle ver- 
tex (Fig. Oi) . Due to the local nature of the auxiliary 
fermions of a bare spin model, a real-space representa- 
tion of all vertices is much more suitable than the usual 
momentum space scheme. Furthermore, since we operate 
in the strong coupling limit, the proper regularization of 
Green's functions requires the self-consistent back cou- 
pling of the self energy (Fig. [5^) into the propagators in 
Fig.[2)D. In order to account for the dynamic fluctuations, 
it is also necessary to keep all frequency dependencies of 
the vertex functions involved. The numerical solution re- 
quires the discretization of these frequencies. Similarly, 
the spatial dependence is approximated by keeping cor- 
relation functions of an infinite system up to a maximal 
length. In our calculations, this length extends over 7 
lattice spacings, leading to a correlation area of 169 lat- 
tice sites (this constrains the resolvable incommensurable 
order to vector sizes within that range). An ordering in- 
stability is initially signalled by a strong rise of the vertex 
couplings associated with this order at some finite scale 
of A (Fig.[3]and|4l). Since in the present formulation rota- 
tional invariance is conserved during the flow, we should 
not find stable solutions down to A = in magnetically 
ordered regimes. Indeed, the onset of spontaneous long 
range order is signalled by a sudden breakdown of the 
smooth flow. In contrast, the existence of a stable solu- 
tion indicates the absence of long range order associated 
with spin rotational symmetry breaking. In the present 
implementation, our approach does not directly detect 



chiral order. This is because the chirality operator is a 
three-body operator by construction, and hence relates 
to the 3-particlc vertex whose explicit RG flow is not 
computed. 

We sweep through the parameter space of ^ from the 
square lattice to the isotropic triangular lattice and com- 
pute the static magnetic susceptibility shown in Fig. [3] 
(the peak positions for different types of long range or- 
der are depicted in Fig. [TJd) . Throughout this parame- 
ter regime, we observe a characteristic breakdown of the 
flow, indicating ordering instabilities rather than a disor- 
dered phase. One can nicely observe how the susceptibil- 
ity evolves as we increase ^. As shown in Fig. [3l we find 
a broadening and subsequent splitting of the Neel-peak. 
The splitted peaks evolves along the Brillouin-zone edge 
and the weight at the corners of the hexagon increases un- 
til in the isotropic case the peak symmetrically resides at 
the corner positions. From the splitting of the Neel peak, 
we estimate the transition to be at ^ci ^ 0.6 — 0.7, i.e., 
the regime above which the order becomes incommen- 
surate. Apparently, the system influenced by quantum 
fluctuation favors AFM fluctuations over spiral fluctua- 
tions, since the classical transition point at ^ = 0.5 is 
shifted to higher ^. In particular, from the extremely 
smooth evolution of the susceptibility upon variation of 
^ which we studied by steps of 0.02 in ^, we find indi- 
cation for the transition to be of second order, while an 
extremely weak first order transition (corresponding to 
a slight kink in the leading susceptibility channel) can- 
not be excluded as a matter of principle. The height of 
the peaks such as in Fig. [3] still depend on system size. 
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FIG. 4. (color online) Static magnetic susceptibility, varying ^ from the isotropic triangular lattice ^ — 1.0 towards the Id chain 
limit ^ — ^ oo. Bottom left: Vertex flow at the ^ = 1 point. The rise of the 120° Neel channel shows the ordering instability at 
the isotropic triangle point. Compared to the flow at ^ = (Fig. |3]), the rise takes place at a much lower scale of A, indicating 
a lower ordering scale for the triangular lattice. 



At the isotropic triangular point ^ = 1 where additional 
lattice symmetries enable us to consider systems even be- 
yond 250 sites, we obtain higher and sharper peaks with 
increasing system size. 

In a recent work, Starykh and Balents have found 
that for ^ > 1, the quantum s yst em enters a collinear 
antiferromagnetic stripe phase [26|. This is interesting 
since the classical estimate would be spiral order in that 
regime, so the quantum fluctuations lead to a different 
ordering [23[. However, finite size numerical studies re- 
ported a disordered phase in that regime [2l|. Our re- 



sults for the magnetic susceptibility for ^ > 1 are shown 
in Fig. m As depicted, we observe a strong drop in the 
magnetic susceptibility above the isotropic point, i.e., in 
the regime ^c2 ^ l-l- From here, no ordering instabil- 
ity is found in the RG flow and the susceptibility rapidly 
looses the 120° Neel order signature. While the 120° Neel 
order peaks die out quickly, AFM stripe fluctuation sig- 
natures emerge (at points of the Brillouin zone according 
to Fig. [T]). The transition appears to be of first order 
according to a pronounced kink in the maximal suscep- 
tibility upon varying ^. While we do not find a break- 
down of the flow that would indicate magnetic ordering, 
we still obtain stron g co llinear AFM stripe fluctuations 
(in agreement with [26|) signalled by an unstable RG 
flow that develops oscillations sensitively depending on 
the frequency discretization. These fluctuation tenden- 
cies are also seen at higher ^ where the peak structure is 
still visible along the fca;-direction. However, these peaks 
are strongly broadened along the fcy-axis, i.e., smeared 
between the two c-AFM ordering vector positions. This 
indicates a fast exponential decay of spin correlations be- 



tween the J'-chains. There are proposals in the literature 
that the (supposedly) disordered regime splits into two 
(gapped or gapless) different spin liquid phases In 
principle this would correspond to a change of diffuse 
spectral weight in the magnetic susceptibility going from 
an ungapped to a gapped system. We do not find clear 
indication for this scenario. At large ^ the susceptibility 
does not change significantly with increasing ^, which we 
studied by computing scenarios up to ^ = 5 in steps of 
0.2 in ^. 



In conclusion, we have used the pseudo-fermion func- 
tional renormalization group to study the different phases 
of the anisotropic triangular lattice. We find that upon 
variation of the anisotropy parameter ^, the system di- 
vides into a Neel order, spiral order, and a disordered 
phase with c-AFM stripe fluctuations. We find evidence 
for a second order transition between the first two and a 
first order transition between the last two of these phases. 
We are confident that our method is a suitable starting 
point to discuss various other problems in the field of 
frustrated magnetism. 
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Supplementary material 

Neel order to spiral order transition. In the following 
we have resolved the Neel order to spiral order transition 
in higher resolution varying the anisotropy parameter ^ 
in small stepwidths (see Fig. [5]). We find that the previ- 
ous Neel peak first broadens along kx and then smoothly 
splits up into two peaks which then evolve along the 
Brillouin-zone boundary. This process comes along with 
increasing spectral weight at the corners of the Brillouin 
zone. Note that as a consequence of the periodicity in 
momentum space, the emerging peak structure at ky = 
presents the tails of the broadened Neel peak. Increas- 
ing ^ further towards the isotropic triangular point (see 
Fig. 131) the split peaks move towards the corner position 
until at ^ = 1 the hexagonal symmetry of the suscepti- 
bility is reached. As the susceptibility evolves completely 
smoothly through the transition, we find it to be of sec- 
ond order. We identify the wave vector of the corre- 
sponding long range-ordered phases with the position of 
the maximal susceptibility. From this we locate the tran- 
sition point at such ^ where the peaks split and above 
which the order becomes incommensurate. From Fig. [5] 
a transition at ^ « 0.61 can be read off. In the calcula- 
tions for this figure we kept correlations up to a length 
of 5 lattice spacings (as compared to 7 lattice spacings in 
Fig.[3]and Fig.|3]). In addition, the number of discrete fre- 
quencies has been slightly reduced in Fig. [51 These reduc- 
tions generally result in lower and broader peaks, which 
demonstrates that (according to the fact that the phases 
in this parameter regime are long range-ordered) there 
are still finite size effects in the susceptibility. Especially 
at the isotropic triangular point where additional lattice 
symmetries enable us to treat much longer correlations, 
we are able to resolve peaks at bigger system sizes which, 
in units of Fig. SI reach heights up to 12 in 1/J, while 
the structure of the susceptibility is unchanged. How- 
ever, in order to be able to detect long range order, for 
which the breakdown of the smooth RG-flow is the rele- 
vant criterion, we found it sufficient to use 5 or 7 lattice 
spacings as the maximal length of correlations. Another 
effect seen in a finite size scaling is that with increas- 
ing length of correlations (and also increasing number of 
discrete frequencies) , the transition point moves towards 
slightly higher ^. Therefore from a finite size scaling we 
estimate the transition to be approximately centered be- 
tween ^ = 0.6 and ^ = 0.7. 

High anisotropy regime. We provide more detailed 
data on the strong anisotropy regime (Fig. [S]) . As found 
there, the strong decay of correlations between the effec- 
tively weakly coupled one-dimensional chains is already 
dominant, which manifests itself in a strong smearing of 
the magnetic susceptibility along ky. As ^ is increased, 
the spectral weight of the susceptibility gets more and 
more reduced, while the fluctuation structure remains 
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FIG. 5. Static magnetic susceptibility, varying ^ from the AFM Neel side across the phase transition to spiral ordering. The 
already broadened AFM Neel peak splits up into two incommensurate peaks that successively shift towards the edges of the 
Brillouin zone where the 120° Neel order peaks reside. 




FIG. 6. Static magnetic susceptibility, varying ^ in the strongly anisotropic regime from ^ = 2 to ^ = 4. While the spectral 
structure remains rather invariant of C-AFM type which is strongly smeared along the ky direction, the total spectral weight 
slightly decreases with increasing 



rather stable. We only observe reminiscences of C-AFM 
fluctuations smeared along ky. From there, we do not 
find clear indication for a bi-spin liquid scenario sug- 
gested in the literature for that regime, where one mag- 
netically disordered phase should be gapped while the 
other should be gapless [Yunoki and Sorella, Phys. Rev. 
B 74, 014408 (2006).]. A transition between those phases 
should manifest itself in the susceptibility by some jump 
in the spectral weight. In general, at large anisotropics, 



energy scales involved in the coupling between the chains 
become so small that a caveat has to be given from the 
frequency discretization incorporated in the numerics, 
which sets a lower bound of energy scales to be resolved 
by our method. We varied the frequency mesh in the 
lower energy regime, and found no notable variation of 
our results. While this should hold for the anisotropy 
regime shown in the manuscript, we cannot exclude dif- 
ferent physical scenarios for even higher anisotropy. 



